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ABSTRACT 

The  ubiquity  of  digital  color  image  content  continues  to  raise 
consumer  technological  awareness  and  expectations,  and  places  a 
greater  demand  than  ever  on  algorithms  that  support  color  image  ac¬ 
quisition  for  mobile  devices.  In  this  paper  we  consider  the  key  signal 
processing  challenges  to  advancing  the  digital  camera  pipeline  for 
mobile  multimedia,  with  a  particular  focus  on  advances  that  have  the 
potential  to  enhance  image  quality  and  reduce  overall  cost  and  power 
consumption.  We  first  examine  key  technical  challenges  to  pipeline 
design  presented  by  demands  such  as  shrinking  device  footprints,  in¬ 
creasing  throughput,  and  enhancing  color  fidelity.  We  then  describe 
a  recently  introduced  analytical  framework  based  on  spatio-spectral 
sampling  for  color  image  acquisition,  and  discuss  its  potential  im¬ 
plications  for  quality  and  cost  improvements.  We  then  describe  a 
number  of  resolution-distortion  trade-offs,  in  particular  noise  pro¬ 
cesses  and  crosstalk,  and  show  via  simulation  how  a  spatio-spectral 
acquisition  framework  helps  to  pinpoint  aspects  of  pipeline  design 
that  can  enhance  computational  efficiency  and  performance  simulta¬ 
neously. 

Index  Terms —  image  sensors,  image  sampling,  image  recon¬ 
struction,  image  denoising,  image  color  analysis 


returns  in  terms  of  image  quality,  and  in  our  previous  work  we  pro¬ 
posed  a  novel  acquisition  scheme  that  preserves  the  integrity  of  the 
signal  during  acquisition  [1|.  Shrinking  the  device  footprint  is  an¬ 
other  important  step  toward  increasing  the  pixel  sensor  count  in  a 
cost-effective  manner — a  trend  partly  fueled  by  the  popular  percep¬ 
tion  that  higher  spatial  resolution  necessarily  leads  to  better  image 
quality.  The  device  footprint  reduction  problem  is  complicated  by 
the  problems  such  as  noise  and  crosstalk  that  are  difficult  to  model 
and  quantify  [2-5],  As  the  image  sensor  represents  the  first  step  in 
the  digital  camera  pipeline,  it  largely  determines  the  image  quality 
achievable  by  subsequent  processing  schemes. 

In  this  paper  we  offer  a  signal  processing  framework  for  un¬ 
derstanding  the  resolution-distortion  trade-offs,  its  implications  for 
the  complexity  of  the  subsequent  processing  steps,  and  the  possi¬ 
bilities  for  future  improvements.  Our  analysis  extends  the  spatio- 
spectral  sampling  theory  for  color  image  acquisition  that  provides 
insight  into  the  trade-offs  between  the  effective  resolution  and  de¬ 
gree  of  degradation  due  to  aliasing  [1],  In  particular,  we  present  a 
signal  processing  perspective  on  the  components  of  the  digital  cam¬ 
era  pipeline,  such  as  sampling,  noise,  crosstalk,  and  reconstruction, 
and  evaluate  its  expected  performance  using  the  prior  knowledge  of 
image  signals. 


1.  INTRODUCTION 


2.  BACKGROUND 


With  the  annual  sales  of  mobile  phones  projected  to  exceed  one  bil¬ 
lion  handsets  by  2009,  mobile  multimedia  is  well  positioned  to  be¬ 
come  a  mainstream  platform  for  entertainment.  Despite  narrowing 
profit  margins  and  increased  competition,  the  growing  ubiquity  of 
digital  multimedia  drives  the  demands  for  increased  throughput  and 
improved  image  quality  in  color  imaging  devices.  Low-cost  and  low- 
power  hardware  designs  nevertheless  top  the  list  of  priorities  for  the 
mobile  phone  industry,  and  in  this  respect  signal-processing  solu¬ 
tions  offer  attractive  benefits  and  cost  savings  that  cannot  be  ignored. 

In  this  paper  we  develop  a  signal  processing  perspective  on  the 
future  of  the  digital  camera  pipeline,  with  a  particular  focus  on  the 
advances  that  have  the  potential  to  enhance  image  quality  and  reduce 
overall  cost  and  power  consumption.  Our  interest  lies  in  quantifying 
the  trade-offs  between  performance  and  complexity — and  we  do  so 
not  by  explicit  comparisons  of  digital  camera  pipelines,  but  by  con¬ 
sidering  the  imaging  system's  resilience  to  noise,  aliasing,  and  arti¬ 
facts  that  make  the  subsequent  data  processing  steps  more  compli¬ 
cated  and  expensive.  Given  that  mushrooming  data  rates  pose  addi¬ 
tional  computational,  transmission,  and  storage  challenges  that  can¬ 
not  be  solved  by  the  advances  in  hardware  alone,  this  type  of  analysis 
presents  opportunities  for  new  contributions  in  low-complexity  low- 
cost  digital  camera  designs  through  signal  processing  advancements. 

For  example,  the  inherent  shortcomings  of  color  filter  array  de¬ 
signs  mean  that  subsequent  processing  steps  often  yield  diminishing 


2.1.  Review:  Color  Image  Sensor 

In  this  section,  we  review  components  of  the  image  sensor  that  play 
important  roles  in  determining  overall  limits  in  achievable  image 
quality  relative  to  subsequent  processing  steps.  Let  x  (t)  = 

[xr{t),Xg(t),Xb(t)\  be  the  RGB  tristimulus  value  of  the  desired 
continuous  color  image  signal  at  spatial  location  t  £  R2.  Each 
pixel  sensor  is  equipped  with  a  microlens,  an  optical  MEMS  de¬ 
vice  designed  to  increase  the  fill  factor  (area  of  the  spatial  integra¬ 
tion)  by  locally  focusing  the  light  away  from  circuitries  and  toward 
the  regions  of  the  pixel  sensor  that  are  photosensitive.  Given  a  spa¬ 
tial  sampling  interval  r  for  the  sensor,  the  integrated  light  x(n)  = 
[xr{n),  xg(n),  Xb(n)]  at  pixel  location  n  G  Z2  is: 


x(n) 


{h  *  ir}(rn) 
{h  *  i9}(rn) 
{h  *  Xb}(rn) 


(1) 


where  ‘*’  indicates  convolution  and  h{t)  is  a  filter  that  represents  a 
spatial  integration  over  the  pixel  sensor.  The  photons  collected  by  the 
microlens  must  then  penetrate  through  color  filter  before  reaching 
the  photosensitive  element  of  the  sensor.  CMOS  photo  diode  active 
pixel  sensors  measure  the  intensity  of  the  light  using  a  photo  diode 
and  three  transistors,  all  major  sources  of  noise  [6J.  CCD  sensors,  on 


1  -4244- 1 484-9/08/$25.00  ©2008  IEEE 


5332 


ICASSP  2008 


Report  Documentation  Page 


Form  Approved 
OMB  No.  0704-0188 


Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 


1.  REPORT  DATE 

APR  2008 


2.  REPORT  TYPE 


3.  DATES  COVERED 

00-00-2008  to  00-00-2008 


4.  TITLE  AND  SUBTITLE 

Advancing  the  Digital  Camera  Pipeline  for  Mobile  Multimedia:  Key 
Challenges  from  a  Signal  Processing  Perspective 

6.  AUTHOR(S) 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Harvard  University, School  of  Engineering  and  Applied  Sciences, Oxford 
Street, Cambridge, MA, 02138 


5a.  CONTRACT  NUMBER 


5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

5d.  PROIECT  NUMBER 


5e.  TASK  NUMBER 


5f.  WORK  UNIT  NUMBER 

8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 


9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 


10.  SPONSOR/MONITOR'S  ACRONYM(S) 


11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

See  also  ADM002091.  Presented  at  the  IEEE  International  Conference  on  Acoustics,  Speech,  and  Signal 
Processing  (ICASSP  2008),  Held  in  Las  Vegas,  Nevada  on  March  30-April  4,  2008.  U.S.  Government  or 
Federal  Purpose  Rights  License. 

14.  ABSTRACT 

see  report 

15.  SUBIECT  TERMS 


16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 

ABSTRACT 

18.  NUMBER 

OF  PAGES 

19a.  NAME  OF 

RESPONSIBLE  PERSON 

a.  REPORT 

unclassified 

b.  ABSTRACT 

unclassified 

c.  THIS  PAGE 

unclassified 

Same  as 
Report  (SAR) 

4 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


the  other  hand,  rely  on  the  electron-hole  pair  that  is  generated  when 
a  photon  strikes  silicon  [7]. 

A  color  filter  array  (CFA)  is  a  physical  construction  whereby  the 
spectral  components  of  the  light  are  spatially  multiplexed — that  is, 
each  pixel  location  measures  the  intensity  of  the  light  corresponding 
to  only  a  single  color  [8].  Let  c(n)  = 

[cr(n),cg(n),cb(n)]  represent  the  CFA  color  combination  corre¬ 
sponding  to  x(n ),  and  dr  and  <4  be  the  DC  components  of  cr  and 
Cb-  If  cr  +  c9  +  ct  =  A  for  some  constant  A,  then  light  penetrating 
the  color  filter  may  be  written  as: 


y(n )  =  c(n)T x(n) 


A[ca(n),  1,  cp(n)\ 


xa{n) 
xi{n) 
xp  (n) 


(2) 


where  xa  =  xr  —  xg,  xp  =  Xb  —  xg  are  difference  images,  xe  = 
xg  +  drXa  +  dtXp  is  a  baseband  component,  and  ca  =  cr  —  dr  and 
C/3  =  Cb  —  db  are  modulation  carrier  frequencies  [1],  The  advan¬ 
tage  of  the  {xa,  xe,  xp}  representation  is  the  difference  images  en¬ 
joy  rapid  spectral  decay  away  front  their  center  frequencies,  whereas 
baseband  copy  xe  embodies  the  edge  and  texture  information;  more¬ 
over,  {xa,  xe,  xp}  are  generally  observed  to  be  only  weakly  corre¬ 
lated  L9J. 

While  a  detailed  investigation  of  noise  sources  is  beyond  the 
scope  of  this  paper,  studies  suggest  that  z(n),  the  number  of  pho¬ 
tons  encountered  during  a  spatio-temporal  integration,  is  a  Poisson 
process  denoted  as  z(n)\y(n)  V(k  ■  y(n)),  where  k  is  a  propor¬ 
tionality  constant  that  scales  linearly  with  the  integration  time  and 
surface  area  of  pixels  and  lens.  Note  -E[z|y]  =  ky ,  Var(z|t/)  =  ky, 
and  when  ky  sufficiently  large,  p{z\y)  converges  weakly  to  the  nor¬ 
mal  distribution  A f(ky,  ky) .  In  practice,  the  photo  diode  charge  (e.g. 
photodetector  readout  signal)  is  assumed  proportional  to  z(n),  thus 
we  interpret  ky(n)  and  z(n)  as  the  ideal  and  noisy  sensor  data  at 
pixel  location  n,  respectively. 


2.2.  Review:  Digital  Camera  Pipeline 

Given  the  sensor  data  z(n),  the  goal  of  the  digital  camera  pipeline 
is  to  estimate  the  color  image  x(t) — note  that  we  use  the  continu¬ 
ous  representation  of  the  image  here  in  order  to  better  compare  im¬ 
ages  captured  at  different  sampling  rates.  As  stated  earlier,  one  cost- 
effective  measure  to  increase  spatial  resolution  is  simply  to  shrink 
the  device  footprint  in  hardware.  However,  device  physics  and  sim¬ 
ple  geometric  arguments  (fewer  incident  photons,  for  example)  dic¬ 
tate  that  this  increase  in  spatial  resolution  will  be  accompanied  by 
a  corresponding  increase  in  sensor  noise  effects.  To  understand  this 
trade-off,  we  first  review  a  number  of  signal  processing  steps  or  mod¬ 
ules  that  comprise  a  camera  pipeline  after  the  acquisition  of  data: 

•  The  spatial  subsampling  due  to  the  implementation  of  color 
filter  array  is  approximately  inverted  through  demosaicking — 
yielding  a  complete  tristimulus  value  at  each  pixel  location. 
Assuming  y  (ideal  sensor  data)  as  an  input  to  the  demosaick¬ 
ing  algorithm,  demosaicking  is  a  demultiplexing  of  frequency 
multiplexed  signals  {xa,xe,xp}  [1, 10]. 

•  Because  the  color  coordinates  defined  by  the  sensitivity  of 
the  color  filters  c  may  not  correspond  exactly  to  the  standard¬ 
ized  color  space  (such  as  sRGB  space),  the  resulting  tristim¬ 
ulus  values  undergo  a  color  space  conversion  (change  of  ba¬ 
sis  )  via  pixel-wise  multiplication  by  a  predetermined  matrix 
M  £  R3x3,  xo (n)  =  Mox(n).  Additional  color  space 
conversion  may  be  required  for  image  compression,  which 
usually  operates  in  an  opponent  color  space. 


•  Given  the  variability  of  the  Poisson  process,  the  Poisson  mean 
y  is  often  inaccessible  and  must  be  estimated  from  z.  Alter¬ 
natively,  demosaicking  methods  applied  to  z  as  a  proxy  for  y 
yield  a  “noisy”  estimate  of  x. 

•  Human  visual  systems  make  adjustments  to  the  color  to  ac¬ 
count  for  variations  in  illuminant  and  the  environment.  Linear 
white-balance  correction — needed  to  match  the  camera  out¬ 
put  with  the  perceived  color — is  a  function  of  the  estimated 
scene  illuminant  and  typically  takes  place  either  before  de¬ 
mosaicking  or  concurrently  with  the  color  space  conversion: 
xi(n)  =  lWi(illuminant)a:(n). 

•  A  point-wise  nonlinearity  termed  the  inverse  gamma  func¬ 
tion  r-1  :  R  — >  R  is  applied  to  the  color-corrected  tristimu- 
lus  value  x  to  yield  the  display  stimulus  u  =  [tti,  112,  «3]T, 
Ui{n)  =  r~ 1  {ati (n) }.  This  gamma  correction  step  will 
undo  the  effects  of  nonlinearity  T  inherent  in  display  devices 
(i.e.,  r(u)  is  linear  with  respect  to  x). 

Each  of  the  key  components  in  a  typical  camera  pipeline  is  aimed 
at  correcting  or  enhancing  certain  aspects  of  the  hardware  or  human- 
hardware  interface — and  in  particular,  the  demosaicking,  color  space 
conversion,  and  Poisson  mean  estimation  steps  are  explicitly  coupled 
to  the  image  data  acquisition  process  highlighted  in  the  previous  sec¬ 
tion.  The  key  challenges  for  advancing  the  digital  camera  pipeline 
from  a  signal  processing  perspective,  therefore,  involve  resolution- 
distortion  trade-offs  as  manifested  by  the  color  image  sensor  itself — 
the  subject  of  the  rest  of  this  article. 

3.  RESOLUTION-DISTORTION  TRADE-OFFS 

In  this  section,  we  examine  the  inherent  trade-offs  between  spatial 
resolution  and  signal-dependent  measurement  noise  and  other  dis¬ 
tortions  via  spatio-spectral  sampling  theory  [1J.  The  main  result  of 
our  analysis  is  that  noise  dependency  is  increasingly  severe  as  sensor 
size  decreases. 


3.1.  Resolution  and  Poisson  Process 

For  analytical  tractability,  let  h(t)  be  an  ideal  low-pass  filter.  Com¬ 
bining  (1),  (2),  and  the  effects  of  the  Poisson  process,  the  measure¬ 
ment  z(n)  can  be  characterized  as: 


E[z(n)\x]  =  Var(z(n)|a:)  (3) 

=  k\{h  *  xe}(rn)  +  kvhx(ca(n)xa(Tn)  +  cp(n)xp(rnf\ , 


where  Uh  is  the  DC  component  of  the  convolution  filter  h(t)  and 
for  sufficiently  low-bandwidth  difference  images,  h  *  xa  —  VhXa 
and  h  *  xp  =  o^xp.  From  the  spatio-spectral  sampling  perspec¬ 
tive,  z(n)  is  a  lowpass  version  of  xi  (i.e.,  kX{h  *  ij}(™ ))  that 
has  been  corrupted  by  two  sources  of  degradation:  aliasing  (i.e., 
kXca(n){h  *  Xa}{rn)  +  k\cp(n){h  *  xp}(rn)),  and  the  vari¬ 
ability  resulting  front  the  Poisson  process  (i.e.  Var(z(n)|a:)).  Re¬ 
construction  amounts  to  separating  xe  from  these  interfering  signals 
in  z(n) — without  making  any  additional  assumptions,  such  as  the 
local  sparsity  assumed  by  contemporary  nonlinear  demosaicking  al¬ 
gorithms  [9],  A  little  algebra  will  verify  that  the  distortion  in  z(n) 
relative  to  xe(t)  can  be  measured  as: 


J(x) 


kuhX 


xe(t) 
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where  W{z}  is  the  Whittaker-Shannon  ideal  reconstruction  of  the 
discrete  samples  z(n )  (i.e.  orthogonal  projection  to  space  of  ban- 
dlimited  functions).  The  expectation  E[J(x)\  may  further  be  de¬ 
composed  as 


E 


W{y}(t ) 


xe(t) 


2 


+ 


W{z  -  ky}(t) 
kuhX 


(4) 


Let  E[xa]  =  E[xp\  =  0  and  {xa,xi,  xp]  be  mutually  independent. 
Then  the  first  term  in  (4)  expands  to: 

E[\\{h  *  xg}/vh  -  xe\\2  +  || caxa  +  cpxp\\2\.  (5) 

The  first  term  in  (5)  is  the  degradation  due  to  loss  of  resolution,  and 
it  is  independent  of  the  choice  of  color  filter  array  and  the  surface 
area  of  the  pixel  sensor.  The  second  term  in  (5)  is  the  aliasing  from 
CFA  sampling,  which  is  independent  of  the  surface  area  of  the  pixel 
sensor  and  the  resolution.  Similarly,  the  second  term  in  (4)  is  the 
measurement  noise;  from  (3),  we  see  that  this  is  equivalent  to 

E\yai(z(n)\x) / (kvhXf]  =  {h*  xi}(rn)  /  (ki\X).  (6) 

The  conclusion  we  draw  from  the  above  exercise  is  that  larger 
pixel  sensor  area  (i.e.  larger  k )  and  panchromatic  CFA  pattern  (i.e.  larger 
A)  are  favorable  for  reducing  the  measurement  noise.  The  effects  of 
the  resolution  on  noise  (h  and  Vh)  are  signal  dependent,  though  k 
and  h  are  often  inversely  coupled,  and  the  trade-off  between  (5)  and 
(6)  is  not  straightforward.  The  expected  distortion  E[J(x)\  can  be 
evaluated  empirically  using  simulation — choosing  widely  available 
test  images  for  x,  Figure  1  shows  distortion  as  a  function  of  sensor 
resolution.  A  fixed  value  of  k  means  that  the  overall  image  sensor 
size  and  integration  time  is  held  constant  while  the  sensor  surface  is 
divided  up  into  smaller  pixels  to  increase  resolution.  The  key  ob¬ 
servation  here  is  that  despite  increased  resolution,  shrinking  pixel 
sensors  may  result  in  more  distortion  in  some  cases.  The  problem  is 
especially  bad  when  k  or  x  is  small  (small  sensor  size,  small  lens, 
low-light  environment,  etc).  The  graph  also  suggests  that  a  better 
CFA  design  reduces  distortion  far  more  effectively  than  increasing 
the  pixel  count. 


Fig.  1.  Distortion  with  respect  to  xi  as  a  function  of  the  sensor 
resolution  while  holding  the  sensor  size  constant,  measured  from  a 
simulation  using  24-bit  512  x  768  images.  Solid  lines  indicate  Bayer 
CFA  [8],  dashed  lines  indicate  spatio-spectral  CFA  design  [1], 


Minority  carrier  diffusion  deteriorates  the  signal  when  photons 
stray  from  the  target  after  the  charge  is  collected  [3],  This  carrier  is 
typically  deterministic  and  mostly  linear  with  respect  to  the  signal 
strength,  and  it  can  be  modeled  as  spatially-invariant  convolution: 
z(n )  =  52m  z(n  —  m)g(m),  where  g(m)  is  the  convolution  ker¬ 
nel.  Note  that  the  Poisson  noise  in  i  is  no  longer  spatially  uncor¬ 
related.  Motivated  by  physics,  the  characteristics  of  this  diffusion 
process  are  crudely  modeled  as  g(n)  oc  L,  where  L  is  the 

diffusion  constant  and  r  is  the  sample  interval  [3]. 

Using  the  updated  definitions  y  and  z,  distortion  with  crosstalk 
is  measured  as: 


j[x) 


kvfVgVhX 


3.2.  Resolution  and  Crosstalk 

Crosstalk,  a  phenomenon  where  photon  or  electron  leakages  cause 
an  interaction  between  neighboring  pixels,  is  a  major  problem  when 
the  device  footprint  decreases  because  of  reduced  distances  between 
pixel  sensors.  Two  major  contributions  to  crosstalk  we  consider  here 
are  optical  diffraction  and  minority  carrier  diffusion. 

Optical  diffraction  occurs  when  a  high  incidence  angle  of  the 
light  entering  the  substrate  causes  the  photons  to  stray  away  from 
the  center  of  the  pixel;  microlenses  can  help  to  reduce  this  risk  [2], 
The  diffusion  is  stochastic  but  mostly  linear  with  respect  to  the  in¬ 
tensity  of  the  light.  The  incident  angle  is  typically  wider  for  the 
pixel  sensors  far  from  the  lens  axis,  and  thus  the  light  that  reaches 
photosensitive  material  can  be  modeled  as  spatially-variant  convo¬ 
lution:  y(n)  =  52m  2/(m)/(n> m)  where  f(n,  m)  is  the  location- 
dependent  impulse  response.  The  precise  modeling  of  /(n,  m)  as 
a  function  of  sensor  geometry  is  an  active  area  of  research  involving 
sophisticated  simulation  [2,4].  Nevertheless,  location-independent 
approximation  of  the  point-spread-function  /  using  ideal  low-pass 
filters  have  been  suggested  [5],  As  the  coupling  between  pixels  occur 
before  charge  collection,  the  Poisson  noise  is  spatially  uncorrelated, 
so  that  z(n)  V(y{n)). 


where  i //  and  ug  are  the  DC  values  of  the  convolution  filters  f  and  g. 
Breaking  down  into  loss  of  resolution,  aliasing,  and  noise  as  before, 
E[J(x)\  is  equivalent  to  the  sum  of  the  following  terms: 

E[\\{f  *g*h*xe}/{i'fizguh)  -  X£||2], 

E[\\f  *  g  *  {caxa  +  cpxp}\\2],  (7) 

E[g 2  *  {/*  h*  xe}{Tn) / (kujugulX)]. 

As  before,  Figure  2  evaluates  the  expected  distortion  E[J(x)]  em¬ 
pirically  using  test  images.  A  rather  surprising  consequence  of  (7) 
is  that  the  low-pass  convolution  filters  g  and  /  may  help  suppress 
the  distortion  in  z(n)  relative  to  xe(t)  because  they  attenuate  the 
aliasing  components  ( caxa  +  cpxp),  which,  owing  to  the  carrier 
frequencies  ca  and  cp,  occupy  the  high-pass  region. 

The  real  penalty  imposed  by  crosstalk  in  the  trade-off  analysis  is 
the  reconstructibility  of  difference  images  xa  and  xp,  which  roughly 
correspond  to  the  chrominance  of  the  image.  The  reconstruction  of 
xa  and  xp  depends  greatly  on  the  preservation  of  modulated  sig¬ 
nal  caxa  +  cpxp.  Consider,  for  example,  the  amplitude  response 
of  g  at  the  highest  modulation  frequencies  in  ca  and  cp  as  a  func¬ 
tion  of  resolution  (assuming  a  fixed  overall  sensor  area).  Owing  to 
the  rapid  spectral  decay  of  Gaussian  filters,  the  modulated  signal 
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Fig.  2.  Distortion  with  respect  to  xi  as  a  function  of  the  sensor 
resolution  with  crosstalk  artifacts.  Solid  lines  indicate  Bayer  CFA 
[8],  dashed  lines  indicate  spatio-spectral  CFA  design  [1J. 
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4.  DISCUSSION  AND  CONCLUSION 


Motivated  by  the  perspective  that  noise,  aliasing,  and  artifacts  in  an 
imaging  system  lead  to  more  complicated  and  expensive  signal  pro¬ 
cessing  steps  in  digital  camera  pipeline,  we  have  offered  here  a  signal 
processing  perspective  on  trade-offs  between  resolution  and  distor¬ 
tion  as  device  footprints  continue  to  shrink.  We  characterized  the 
color  image  sensor  in  terms  of  physical  properties  such  as  spatio- 
temporal  integration,  color  filter  array,  Poisson  process,  and  elec¬ 
tron/photon  leakage,  and  analytically  and  numerically  evaluated  the 
distortion  in  the  measured  sensor  data.  We  found  that  advantages  to 
shrinking  pixel  sensor  geometries  as  a  means  to  increase  resolution 
in  a  cost-effective  manner  may  be  overridden  by  Poisson  noise  in 
the  signal  measurement  process,  and  that  better  CFA  designs  have 
the  potential  to  reduce  distortion  far  more  effectively.  Our  anal¬ 
ysis  of  resolution-crosstalk  trade-offs  revealed  the  mechanism  by 
which  crosstalk  desaturates  the  colors  while  sometimes  improving 
estimates  for  the  luminance  component. 
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